Skip to content

GH-36593: [Python] Add rename_columns method to pyarrow datasets #48289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
JonatanMartens wants to merge 6 commits into
base: main
Choose a base branch
from
Open

GH-36593: [Python] Add rename_columns method to pyarrow datasets #48289

JonatanMartens wants to merge 6 commits into from

Conversation

@JonatanMartens
Copy link

@JonatanMartens JonatanMartens commented Nov 29, 2025 •
edited by github-actions bot
Loading

Rationale for this change

See #36593
In particular this change is convenient when the column names stored in a file are different from the logical names associated with the columns (see deltalake column mapping feature as an example).

What changes are included in this PR?

Adds the rename_columns method to datasets in pyarrow.
This mehod allows a user to rename columns in the data returned from a scan before actually creating a scanner object.

Are these changes tested?

This PR also add a test for the new rename_columns method using an InMemoryDataset.

Are there any user-facing changes?

Adds the rename_columns method to pyarrow datasets.

@github-actions
Copy link

github-actions bot commented Nov 29, 2025

⚠️ GitHub issue #36593 has been automatically assigned in GitHub to PR creator.

raulcd
raulcd reviewed Dec 4, 2025
View reviewed changes
Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the examples where bad you just have to add the imports for doctest to work:
>>> import pyarrow.dataset as ds as seen here:

arrow/python/pyarrow/_dataset.pyx

Lines 371 to 391 in b2e8f25

>>> import pyarrow as pa
>>> table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021],
... 'n_legs': [2, 2, 4, 4, 5, 100],
... 'animal': ["Flamingo", "Parrot", "Dog", "Horse",
... "Brittle stars", "Centipede"]})
>>>
>>> import pyarrow.parquet as pq
>>> pq.write_table(table, "dataset_scanner.parquet")
>>> import pyarrow.dataset as ds
>>> dataset = ds.dataset("dataset_scanner.parquet")
Selecting a subset of the columns:
>>> dataset.scanner(columns=["year", "n_legs"]).to_table()
pyarrow.Table
year: int64
n_legs: int64
----
year: [[2020,2022,2021,2022,2019,2021]]
n_legs: [[2,2,4,4,5,100]]

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Dec 4, 2025
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 4, 2025
@JonatanMartens JonatanMartens mentioned this pull request Dec 4, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raulcd raulcd raulcd left review comments

@AlenkaF AlenkaF Awaiting requested review from AlenkaF AlenkaF is a code owner

@rok rok Awaiting requested review from rok rok is a code owner

Assignees

No one assigned

Labels

awaiting change review Awaiting change review Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@JonatanMartens @raulcd